Adaptation of a language model may be based on a user's historic inputs to a speech recognition algorithm. Conventional language model adaptation has been used in in desktop dictation and other applications that can accumulate a significant amount of user historic voice input and related text, with or without supervision. However, this approach allows for only limited accuracy gain in tasks where large amounts of prior history are unavailable, such as voice search tasks.